27 research outputs found
Online Selection of CMA-ES Variants
In the field of evolutionary computation, one of the most challenging topics
is algorithm selection. Knowing which heuristics to use for which optimization
problem is key to obtaining high-quality solutions. We aim to extend this
research topic by taking a first step towards a selection method for adaptive
CMA-ES algorithms. We build upon the theoretical work done by van Rijn
\textit{et al.} [PPSN'18], in which the potential of switching between
different CMA-ES variants was quantified in the context of a modular CMA-ES
framework.
We demonstrate in this work that their proposed approach is not very
reliable, in that implementing the suggested adaptive configurations does not
yield the predicted performance gains. We propose a revised approach, which
results in a more robust fit between predicted and actual performance. The
adaptive CMA-ES approach obtains performance gains on 18 out of 24 tested
functions of the BBOB benchmark, with stable advantages of up to 23\%. An
analysis of module activation indicates which modules are most crucial for the
different phases of optimizing each of the 24 benchmark problems. The module
activation also suggests that additional gains are possible when including the
(B)IPOP modules, which we have excluded for this present work.Comment: To appear at Genetic and Evolutionary Computation Conference
(GECCO'19) Appendix will be added in due tim
Towards Dynamic Algorithm Selection for Numerical Black-Box Optimization: Investigating BBOB as a Use Case
One of the most challenging problems in evolutionary computation is to select
from its family of diverse solvers one that performs well on a given problem.
This algorithm selection problem is complicated by the fact that different
phases of the optimization process require different search behavior. While
this can partly be controlled by the algorithm itself, there exist large
differences between algorithm performance. It can therefore be beneficial to
swap the configuration or even the entire algorithm during the run. Long deemed
impractical, recent advances in Machine Learning and in exploratory landscape
analysis give hope that this dynamic algorithm configuration~(dynAC) can
eventually be solved by automatically trained configuration schedules. With
this work we aim at promoting research on dynAC, by introducing a simpler
variant that focuses only on switching between different algorithms, not
configurations. Using the rich data from the Black Box Optimization
Benchmark~(BBOB) platform, we show that even single-switch dynamic Algorithm
selection (dynAS) can potentially result in significant performance gains. We
also discuss key challenges in dynAS, and argue that the BBOB-framework can
become a useful tool in overcoming these
Sequential vs. Integrated Algorithm Selection and Configuration: A Case Study for the Modular CMA-ES
When faced with a specific optimization problem, choosing which algorithm to
use is always a tough task. Not only is there a vast variety of algorithms to
select from, but these algorithms often are controlled by many hyperparameters,
which need to be tuned in order to achieve the best performance possible.
Usually, this problem is separated into two parts: algorithm selection and
algorithm configuration. With the significant advances made in Machine
Learning, however, these problems can be integrated into a combined algorithm
selection and hyperparameter optimization task, commonly known as the CASH
problem. In this work we compare sequential and integrated algorithm selection
and configuration approaches for the case of selecting and tuning the best out
of 4608 variants of the Covariance Matrix Adaptation Evolution Strategy
(CMA-ES) tested on the Black Box Optimization Benchmark (BBOB) suite. We first
show that the ranking of the modular CMA-ES variants depends to a large extent
on the quality of the hyperparameters. This implies that even a sequential
approach based on complete enumeration of the algorithm space will likely
result in sub-optimal solutions. In fact, we show that the integrated approach
manages to provide competitive results at a much smaller computational cost. We
also compare two different mixed-integer algorithm configuration techniques,
called irace and Mixed-Integer Parallel Efficient Global Optimization
(MIP-EGO). While we show that the two methods differ significantly in their
treatment of the exploration-exploitation balance, their overall performances
are very similar
Modular Differential Evolution
New contributions in the field of iterative optimisation heuristics are often
made in an iterative manner. Novel algorithmic ideas are not proposed in
isolation, but usually as an extension of a preexisting algorithm. Although
these contributions are often compared to the base algorithm, it is challenging
to make fair comparisons between larger sets of algorithm variants. This
happens because even small changes in the experimental setup, parameter
settings, or implementation details can cause results to become incomparable.
Modular algorithms offer a way to overcome these challenges. By implementing
the algorithmic modifications into a common framework, many algorithm variants
can be compared, while ensuring that implementation details match in all
versions.
In this work, we propose a version of a modular framework for the popular
Differential Evolution (DE) algorithm. We show that this modular approach not
only aids in comparison, but also allows for a much more detailed exploration
of the space of possible DE variants. This is illustrated by showing that
tuning the settings of modular DE vastly outperforms a set of commonly used DE
versions which have been recreated in our framework. We then investigate these
tuned algorithms in detail, highlighting the relation between modules and
performance on particular problems
IOHanalyzer: Performance Analysis for Iterative Optimization Heuristic
Benchmarking and performance analysis play an important role in understanding
the behaviour of iterative optimization heuristics (IOHs) such as local search
algorithms, genetic and evolutionary algorithms, Bayesian optimization
algorithms, etc. This task, however, involves manual setup, execution, and
analysis of the experiment on an individual basis, which is laborious and can
be mitigated by a generic and well-designed platform. For this purpose, we
propose IOHanalyzer, a new user-friendly tool for the analysis, comparison, and
visualization of performance data of IOHs.
Implemented in R and C++, IOHanalyzer is fully open source. It is available
on CRAN and GitHub. IOHanalyzer provides detailed statistics about fixed-target
running times and about fixed-budget performance of the benchmarked algorithms
on real-valued, single-objective optimization tasks. Performance aggregation
over several benchmark problems is possible, for example in the form of
empirical cumulative distribution functions. Key advantages of IOHanalyzer over
other performance analysis packages are its highly interactive design, which
allows users to specify the performance measures, ranges, and granularity that
are most useful for their experiments, and the possibility to analyze not only
performance traces, but also the evolution of dynamic state parameters.
IOHanalyzer can directly process performance data from the main benchmarking
platforms, including the COCO platform, Nevergrad, and our own IOHexperimenter.
An R programming interface is provided for users preferring to have a finer
control over the implemented functionalities
Analysis of modular CMA-ES on strict box-constrained problems in the SBOX-COST benchmarking suite
Box-constraints limit the domain of decision variables and are common in
real-world optimization problems, for example, due to physical, natural or
spatial limitations. Consequently, solutions violating a box-constraint may not
be evaluable. This assumption is often ignored in the literature, e.g.,
existing benchmark suites, such as COCO/BBOB, allow the optimizer to evaluate
infeasible solutions. This paper presents an initial study on the
strict-box-constrained benchmarking suite (SBOX-COST), which is a variant of
the well-known BBOB benchmark suite that enforces box-constraints by returning
an invalid evaluation value for infeasible solutions. Specifically, we want to
understand the performance difference between BBOB and SBOX-COST as a function
of two initialization methods and six constraint-handling strategies all tested
with modular CMA-ES. We find that, contrary to what may be expected, handling
box-constraints by saturation is not always better than not handling them at
all. However, across all BBOB functions, saturation is better than not
handling, and the difference increases with the number of dimensions. Strictly
enforcing box-constraints also has a clear negative effect on the performance
of classical CMA-ES (with uniform random initialization and no constraint
handling), especially as problem dimensionality increases
OPTION: OPTImization Algorithm Benchmarking ONtology
Many optimization algorithm benchmarking platforms allow users to share their
experimental data to promote reproducible and reusable research. However,
different platforms use different data models and formats, which drastically
complicates the identification of relevant datasets, their interpretation, and
their interoperability. Therefore, a semantically rich, ontology-based,
machine-readable data model that can be used by different platforms is highly
desirable. In this paper, we report on the development of such an ontology,
which we call OPTION (OPTImization algorithm benchmarking ONtology). Our
ontology provides the vocabulary needed for semantic annotation of the core
entities involved in the benchmarking process, such as algorithms, problems,
and evaluation measures. It also provides means for automatic data integration,
improved interoperability, and powerful querying capabilities, thereby
increasing the value of the benchmarking data. We demonstrate the utility of
OPTION, by annotating and querying a corpus of benchmark performance data from
the BBOB collection of the COCO framework and from the Yet Another Black-Box
Optimization Benchmark (YABBOB) family of the Nevergrad environment. In
addition, we integrate features of the BBOB functional performance landscape
into the OPTION knowledge base using publicly available datasets with
exploratory landscape analysis. Finally, we integrate the OPTION knowledge base
into the IOHprofiler environment and provide users with the ability to perform
meta-analysis of performance data
Computing Star Discrepancies with Numerical Black-Box Optimization Algorithms
The star discrepancy is a measure for the regularity of a finite
set of points taken from . Low discrepancy point sets are highly
relevant for Quasi-Monte Carlo methods in numerical integration and several
other applications. Unfortunately, computing the star discrepancy
of a given point set is known to be a hard problem, with the best exact
algorithms falling short for even moderate dimensions around 8. However,
despite the difficulty of finding the global maximum that defines the
star discrepancy of the set, local evaluations at selected points
are inexpensive. This makes the problem tractable by black-box optimization
approaches.
In this work we compare 8 popular numerical black-box optimization algorithms
on the star discrepancy computation problem, using a wide set of
instances in dimensions 2 to 15. We show that all used optimizers perform very
badly on a large majority of the instances and that in many cases random search
outperforms even the more sophisticated solvers. We suspect that
state-of-the-art numerical black-box optimization techniques fail to capture
the global structure of the problem, an important shortcoming that may guide
their future development.
We also provide a parallel implementation of the best-known algorithm to
compute the discrepancy.Comment: To appear in the Proceedings of GECCO 202